Displaying a scanned image of a physical note

ABSTRACT

Approaches presented herein enable transferring a physical note to an electronic medium. The approach includes scanning, as an image, a physical note positioned on a display screen of an electronic device and automatically inserting the scanned image into an application being used on the electronic device. A physical note is positioned on a display screen of an electronic device by a user. The physical note is scanned as an image and a location of the physical note is logged. A target application is determined based on the location of the physical note and the scanned image is automatically inserted into the application.

The present patent document is a continuation of U.S. patent applicationSer. No. 15/944,932, filed Apr. 4, 2018, entitled “DISPLAYING A SCANNEDIMAGE OF A PHYSICAL NOTE”, the disclosure of which is incorporatedherein by reference.

TECHNICAL FIELD

The subject matter of this invention relates generally to imageprocessing. More specifically, aspects of the present invention providea solution for transferring a physical note to an electronic medium.

BACKGROUND

Physical notes including sticky-back notes, such as Post-it® notes, arepopularly used for a wide variety of purposes. Sticky-back notes appearon walls and billboards, cluster around computer monitor frames, and areembedded within documents. They can be used for gathering informalfeedback and annotating discussions, for personal notes, asynchronousdiscussion among groups or between persons, and for error correction.Posterboards covered with many colors of sticky-back notes can be usedto document tasks as diverse as brainstorming sessions, projectplanning, and design presentations. Studies on work process for manykinds of designers show that sticky-back notes are ubiquitous.Architects and product designers scribble notes and stick them ontoblueprints or even onto physical models. New media or product designerscan use them to represent nonlinear or sequential content units,enjoying the ease with which events can be rearranged in a storyboard,for example. The ease of use of sticky-back notes is one of theirprincipal advantages. However, the ability to save, track, and searchthe information on sticky-back and other types of physical notes (e.g.,on paper) can be unwieldy. A person may resort to camera capture ofwhiteboards or poster boards covered with physical notes, or topost-session computer entry by hand in order to save the informationcontained within the notes.

SUMMARY

In general, embodiments of the present invention enable scanning, as animage, a physical note positioned on a display screen of an electronicdevice and automatically inserting the scanned image into an applicationbeing used on the electronic device. A physical note is positioned on adisplay screen of an electronic device by a user. The physical note isscanned as an image and a location of the physical note is logged. Atarget application is determined based on the location of the physicalnote and the scanned image is automatically inserted into theapplication.

One aspect of the present invention includes a method for transferring aphysical note to an electronic medium, the method comprising: scanning,as an image, a physical note positioned on a display screen; logging,relative to the display screen, a location of the physical note;identifying, based on the logged location, a target application toinsert the scanned image; and automatically inserting the scanned imageinto the target application.

Another aspect of the present invention includes a computer programproduct embodied in a computer readable medium that, when executed by acomputer device, performs a method for transferring a physical note toan electronic medium, the method comprising: scanning, as an image, aphysical note positioned on a display screen; logging, relative to thedisplay screen, a location of the physical note; identifying, based onthe logged location, a target application to insert the scanned image;and automatically inserting the scanned image into the targetapplication.

Yet another aspect of the present invention includes system fortransferring a physical note to an electronic medium, comprising: adisplay screen having a plurality of pixels, each of the plurality ofpixels having a photodiode and a screen light-emitting diode; a memorymedium comprising instructions; a bus coupled to the memory medium; anda processor coupled to the bus that when executing the instructionscauses the system to perform a method, comprising: scanning, with theplurality of pixels as an image, a physical note positioned on thedisplay screen; logging, relative to the display screen, a location ofthe physical note; identifying, based on the logged location, a targetapplication to insert the scanned image; and automatically inserting thescanned image into the target application.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

These and other features of this invention will be more readilyunderstood from the following detailed description of the variousaspects of the invention taken in conjunction with the accompanyingdrawings in which:

FIG. 1 shows an architecture 10 in which the invention may beimplemented according to an illustrative embodiment of the presentinvention;

FIG. 2 shows a first schematic diagram 200 illustrating an exemplaryenvironment for implementation according to an illustrative embodimentof the present invention;

FIG. 3 shows an example pixel 300 of a display screen according to anillustrative embodiment of the present invention;

FIGS. 4A-B show an example of an increase of a brightness of specificpixels on a display screen according to an illustrative embodiment ofthe present invention;

FIG. 5 shows an example color contrast matrix related to two pixels on adisplay screen according to an illustrative embodiment of the presentinvention;

FIG. 6 shows an example scanned image according to an illustrativeembodiment of the present invention; and

FIG. 7 shows a process flowchart 700 for scanning, as an image, aphysical note positioned on a display screen of an electronic device andautomatically inserting the scanned image into an application of theelectronic device according to an illustrative embodiment of the presentinvention.

The drawings are not necessarily to scale. The drawings are merelyrepresentations, not intended to portray specific parameters of theinvention. The drawings are intended to depict only typical embodimentsof the invention, and therefore should not be considered as limiting inscope. In the drawings, like numbering represents like elements.

DETAILED DESCRIPTION

Illustrative embodiments will now be described more fully herein withreference to the accompanying drawings, in which illustrativeembodiments are shown. It will be appreciated that this disclosure maybe embodied in many different forms and should not be construed aslimited to the illustrative embodiments set forth herein. Rather, theseembodiments are provided so that this disclosure will be thorough andcomplete and will fully convey the scope of this disclosure to thoseskilled in the art.

Furthermore, the terminology used herein is for the purpose ofdescribing particular embodiments only and is not intended to belimiting of this disclosure. As used herein, the singular forms “a”,“an”, and “the” are intended to include the plural forms as well, unlessthe context clearly indicates otherwise. Furthermore, the use of theterms “a”, “an”, etc., do not denote a limitation of quantity, butrather denote the presence of at least one of the referenced items.Furthermore, similar elements in different figures may be assignedsimilar element numbers. It will be further understood that the terms“comprises” and/or “comprising”, or “includes” and/or “including”, whenused in this specification, specify the presence of stated features,regions, integers, steps, operations, elements, and/or components, butdo not preclude the presence or addition of one or more other features,regions, integers, steps, operations, elements, components, and/orgroups thereof.

Unless specifically stated otherwise, it may be appreciated that termssuch as “processing”, “detecting”, “determining”, “evaluating”,“receiving”, or the like, refer to the action and/or processes of acomputer or computing system, or similar electronic data center device,that manipulates and/or transforms data represented as physicalquantities (e.g., electronic) within the computing system's registersand/or memories into other data similarly represented as physicalquantities within the computing system's memories, registers or othersuch information storage, transmission, or viewing devices. Theembodiments are not limited in this context. As used herein, the terms“user” and “consumer” are used interchangeably and refer to anindividual who has opted in to receiving local inventory notifications.

As stated above, embodiments of the present invention enable scanning,as an image, a physical note positioned on a display screen of anelectronic device and automatically inserting the scanned image into anapplication being used on the electronic device. A physical note ispositioned on a display screen of an electronic device by a user. Thephysical note is scanned as an image and a location of the physical noteis logged. A target application is determined based on the location ofthe physical note and the scanned image is automatically inserted intothe application.

For different reasons, taking notes can be an important part of both aperson's personal and professional life. For example, when reading orlistening, taking notes can help a person concentrate. In order to takeuseful notes (e.g., to write something sensible), a person mustunderstand the text. As listening and reading are interactive tasks,taking notes can help one make sense of the text. Taking notes does notmean writing down every word you hear. It requires actively decidingwhat is important and how it is related to what has already been writtenin the notes. Also, notes can help a person maintain a permanent recordof what has been read or listened to. This can be useful when a needarises to refer to the notes in the future, such as when studying for anupcoming exam in school or having important information on hand for abusiness meeting or project.

While writing things out by hand has been shown to help with retention,it can be difficult to keep notes organized. The easiest way for aperson to keep her notes organized is to keep them in one place.However, this can develop into a person keeping a random pile of stickynotes or scraps of paper on a desk which may fall off the desk andbecome lost. To avoid losing them, one can transfer them to a devicesuch as a computer by either manually typing them into the device usinga word processing application or by scanning them onto the device usinga scanner, taking a photo of the notes using a camera and transmittingthe photo to the device. Each of these methods, however, are inefficientand time-consuming.

In addition, sharing of physical notes can be problematic, such asduring a conference call which necessitates the notes being transferredto a device. As mentioned, the transfer of physical notes to a device(e.g., via manual input, scanner, camera, etc.) can be slow and tedious.The approaches described herein provide a seamless way for scanning, asan image, a physical note positioned on a display screen of anelectronic device and automatically inserting the scanned image into anapplication being used on the electronic device.

In certain embodiments, an advantage of this approach is its improvementof note-taking by allowing a person to more easily capture, organize,and share notes. This approach allows handwritten notes to betransitioned seamlessly to digital content without requiring manualinput or the use of an external device (e.g., a scanner or camera),thereby reducing overhead and inefficiencies in producing organizednotes which can be easily accessed and searched. This approach combinesthe advantages of handwriting and digital note-taking while erasing manyof their respective drawbacks.

Referring now to FIG. 1, a computerized implementation 10 of anembodiment for scanning, as an image, a physical note positioned on adisplay screen of an electronic device and automatically inserting thescanned image into an application being used on the electronic devicewill be shown and described. Computerized implementation 10 is only oneexample of a suitable implementation and is not intended to suggest anylimitation as to the scope of use or functionality of embodiments of theinvention described herein. Regardless, computerized implementation 10is capable of being implemented and/or performing any of thefunctionality set forth hereinabove.

In computerized implementation 10, there is a computer system/server 12,which is operational with numerous other general purpose or specialpurpose computing system environments or configurations. Examples ofwell-known computing systems, environments, and/or configurations thatmay be suitable for use with computer system/server 12 include, but arenot limited to, personal computer systems, server computer systems, thinclients, thick clients, hand-held or laptop devices, multiprocessorsystems, microprocessor-based systems, set top boxes, programmableconsumer electronics, network PCs, minicomputer systems, mainframecomputer systems, and distributed cloud computing environments thatinclude any of the above systems or devices, and the like.

This is intended to demonstrate, among other things, that the presentinvention could be implemented within a network environment (e.g., theInternet, a wide area network (WAN), a local area network (LAN), avirtual private network (VPN), etc.), a cloud computing environment, acellular network, or on a stand-alone computer system. Communicationthroughout the network can occur via any combination of various types ofcommunication links. For example, the communication links can compriseaddressable connections that may utilize any combination of wired and/orwireless transmission methods. Where communications occur via theInternet, connectivity could be provided by conventional TCP/IPsockets-based protocol, and an Internet service provider could be usedto establish connectivity to the Internet. Still yet, computersystem/server 12 is intended to demonstrate that some or all of thecomponents of implementation 10 could be deployed, managed, serviced,etc., by a service provider who offers to implement, deploy, and/orperform the functions of the present invention for others.

Computer system/server 12 is intended to represent any type of computersystem that may be implemented in deploying/realizing the teachingsrecited herein. Computer system/server 12 may be described in thegeneral context of computer system executable instructions, such asprogram modules, being executed by a computer system. Generally, programmodules may include routines, programs, objects, components, logic, datastructures, and so on, that perform particular tasks or implementparticular abstract data types. In this particular example, computersystem/server 12 represents an illustrative system for scanning, as animage, a physical note positioned on a display screen of an electronicdevice and automatically inserting the scanned image into an applicationbeing used on the electronic device. It should be understood that anyother computers implemented under the present invention may havedifferent components/software, but can perform similar functions.

Computer system/server 12 in computerized implementation 10 is shown inthe form of a general-purpose computing device. The components ofcomputer system/server 12 may include, but are not limited to, one ormore processors or processing units 16, a system memory 28, and a bus 18that couples various system components including system memory 28 toprocessor 16.

Bus 18 represents one or more of any of several types of bus structures,including a memory bus or memory controller, a peripheral bus, anaccelerated graphics port, and a processor or local bus using any of avariety of bus architectures. By way of example, and not limitation,such architectures include Industry Standard Architecture (ISA) bus,Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnects (PCI) bus.

Processing unit 16 refers, generally, to any apparatus that performslogic operations, computational tasks, control functions, etc. Aprocessor may include one or more subsystems, components, and/or otherprocessors. A processor will typically include various logic componentsthat operate using a clock signal to latch data, advance logic states,synchronize computations and logic operations, and/or provide othertiming functions. During operation, processing unit 16 collects androutes signals representing inputs and outputs between external devices14 and input devices (not shown). The signals can be transmitted over aLAN and/or a WAN (e.g., T1, T3, 56 kb, X.25), broadband connections(ISDN, Frame Relay, ATM), wireless links (802.11, Bluetooth, etc.), andso on. In some embodiments, the signals may be encrypted using, forexample, trusted key-pair encryption. Different systems may transmitinformation using different communication pathways, such as Ethernet orwireless networks, direct serial or parallel connections, USB,Firewire®, Bluetooth®, or other proprietary interfaces. (Firewire is aregistered trademark of Apple Computer, Inc. Bluetooth is a registeredtrademark of Bluetooth Special Interest Group (SIG)).

In general, processing unit 16 executes computer program code, such asprogram code for scanning, as an image, a physical note positioned on adisplay screen of an electronic device and automatically inserting thescanned image into an application being used on the electronic device,which is stored in memory 28, storage system 34, and/or program/utility40. While executing computer program code, processing unit 16 can readand/or write data to/from memory 28, storage system 34, andprogram/utility 40.

Computer system/server 12 typically includes a variety of computersystem readable media. Such media may be any available media that isaccessible by computer system/server 12, and it includes both volatileand non-volatile media, removable and non-removable media.

System memory 28 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 30 and/or cachememory 32. Computer system/server 12 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia, (e.g., VCRs, DVRs, RAID arrays, USB hard drives, optical diskrecorders, flash storage devices, and/or any other data processing andstorage elements for storing and/or processing data). By way of exampleonly, storage system 34 can be provided for reading from and writing toa non-removable, non-volatile magnetic media (not shown and typicallycalled a “hard drive”). Although not shown, a magnetic disk drive forreading from and writing to a removable, non-volatile magnetic disk(e.g., a “floppy disk”), and an optical disk drive for reading from orwriting to a removable, non-volatile optical disk such as a CD-ROM,DVD-ROM, or other optical media can be provided. In such instances, eachcan be connected to bus 18 by one or more data media interfaces. As willbe further depicted and described below, memory 28 may include at leastone program product having a set (e.g., at least one) of program modulesthat are configured to carry out the functions of embodiments of theinvention.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium including, but not limited to, wireless,wireline, optical fiber cable, radio-frequency (RF), etc., or anysuitable combination of the foregoing.

Program/utility 40, having a set (at least one) of program modules 42,may be stored in memory 28 by way of example, and not limitation. Memory28 may also have an operating system, one or more application programs,other program modules, and program data. Each of the operating system,one or more application programs, other program modules, and programdata or some combination thereof, may include an implementation of anetworking environment. Program modules 42 generally carry out thefunctions and/or methodologies of embodiments of the invention asdescribed herein.

Computer system/server 12 may also communicate with one or more externaldevices 14 such as a keyboard, a pointing device, a display 24, etc.;one or more devices that enable a consumer to interact with computersystem/server 12; and/or any devices (e.g., network card, modem, etc.)that enable computer system/server 12 to communicate with one or moreother computing devices. Such communication can occur via I/O interfaces22. Still yet, computer system/server 12 can communicate with one ormore networks such as a local area network (LAN), a general wide areanetwork (WAN), and/or a public network (e.g., the Internet) via networkadapter 20. As depicted, network adapter 20 communicates with the othercomponents of computer system/server 12 via bus 18. It should beunderstood that although not shown, other hardware and/or softwarecomponents could be used in conjunction with computer system/server 12.Examples include, but are not limited to: microcode, device drivers,redundant processing units, external disk drive arrays, RAID systems,tape drives, and data archival storage systems, etc.

Referring now to FIG. 2, a block diagram 200 describing thefunctionality discussed herein according to an embodiment of the presentinvention is shown. It is understood that the teachings recited hereinmay be practiced within any type of computing environment (e.g.,computer system 12). To this extent, the teachings recited herein may bepracticed within a stand-alone computer system or within a networkedcomputing environment (e.g., a client-server environment, peer-to-peerenvironment, distributed computing environment, cloud computingenvironment, and/or the like). If the teachings recited herein arepracticed within a networked computing environment, each electronicdevice need not have a notes capture mechanism 50 (hereinafter “system50”). Rather, system 50 could be loaded on a server or server-capabledevice that communicates (e.g., wirelessly) with the electronic devicefor scanning, as an image, a physical note positioned on a displayscreen of an electronic device and automatically inserting the scannedimage into an application being used on the electronic device. In atypical embodiment, however, system 50 or components of system 50 mayreside within a screen connected to an electronic device capable ofperforming the functions described herein.

Regardless, as depicted, system 50 can be implemented as program/utility40 on computer system 12 of FIG. 1 and can enable the functions recitedherein. It is further understood that system 50 can be incorporatedwithin or work in conjunction with any type of system that receives,processes, and/or executes commands with respect to IT resources in anetworked computing environment. Such other system(s) have not beenshown in FIG. 2 for brevity purposes. As shown, notes capture mechanism50 includes capture component 52, application component 54, andpresentation component 56. The functions/acts of each component isdescribed in detail below.

Capture component 52 of system 50, as executed by computer system/server12, is configured to scan, as an image, a physical note positioned on adisplay screen of an electronic device. In an embodiment, the displayscreen of the electronic device utilizes organic light-emitting diode(OLED) technology, or any other screen technology that is able to turnoff individual pixels without turning off the whole display screen. Asillustrated, FIG. 3 shows an individual pixel 300 of an example displayscreen. Pixel 300 includes screen surface 305, photodiode 310, andscreen light-emitting diode (LED) 315. To be scanned, a physical note ispositioned on a screen surface 305 of the display screen. The physicalnote may be held to the screen by a user or “stuck” to the screen (e.g.,a sticky-note). In any event, the physical note is positioned on thedisplay screen to produce a scanned image of the note.

Capture component 52 may increase the brightness of specific pixelswithin an area associated with the physical note and collect lightinginformation reflecting from the physical note to construct a scannedimage of the note. As shown, LED 315 exists alongside photodiode 310. Asa result of this, when a note is detected, capture component 52 willincrease a brightness of LED 315 allowing photodiode 310 to record thelight reflected back from the note in order to capture a contents of thenote. FIGS. 4A-B show an example on how the display screen can increasea brightness of specific pixels to facilitate scanning a physical noteplaced on the display screen. The process is described in greater detailbelow.

Prior to scanning, however, capture component 52 must ascertain the areaof the display screen to be scanned, either automatically or manually.In an embodiment, capture component 52 may attempt to automaticallydetect an area where the physical note is placed when the display screenis a resistive touch screen. A resistive touch screen may consist of anumber of layers. When the screen is pressed, the outer later is pushedonto the next layer. The technology senses that pressure is beingapplied and registers input. The input can be used to determine the areaof the physical note on the physical screen.

In another embodiment, a calibration process can be performed,regardless of display screen type, to calibrate the display screen toits surroundings in order to measure a difference in a lighting level sothat when a physical note is placed on the screen, photodiode 310 candetect any change in light intensity. If there is a change in lightintensity (e.g., below or above a predefined threshold) before and afterthe physical note is positioned on the display screen, then a change inlight intensity of surrounding pixels can be checked. If the change issimilar or constant, then it can be determined that an object (e.g., aphysical note) was placed on the display screen. The scanning processcan be performed on those pixels experiencing a change in lightintensity. If all of the pixels (or a percentage exceeding a predefinedthreshold) on the display screen experienced a similar change in lightintensity, then it is likely a change occurred in the lightingconditions of the surroundings. In that case, the calibration processcan be performed again to measure the new lighting of the surroundings.

In another embodiment, a user may manually provide an area to be scannedwhen a physical note is held to a display screen. For example, a usercan “outline” with her finger the area to be scanned when the displayscreen is a capacitive touch screen. A capacitive touch screen panelconsists of an insulator, such as glass, coated with a transparentconductor, such as indium tin oxide (ITO). As the human body is also anelectrical conductor, touching the surface of the screen results in adistortion of the screen's electrostatic field, measurable as a changein capacitance. Different technologies may be used to determine thelocation of the touch. Capture component 52 may receive the area basedon the user's outline. Similarly, a user may use her mouse to outlinethe area of the display screen related to the note if the display screenis not a touch screen. Likewise, a manual process can be followed for aresistive touch screen if the physical note is not automaticallyrecognized due to poor lighting conditions or poor calibration. In anycase, an area associated with the note is determined, eitherautomatically or manually, and the scanning process described below isperformed based on the area.

Once the area where the physical note has been placed is determined, thearea can be scanned to generate an image of the physical note that hasbeen placed on the display screen. FIG. 5 shows an example colorcontrast matrix related to two pixels on a display screen (i.e., Pixel605 and Pixel 610 of FIG. 6). At 505, the two pixels are shown after thecalibration process. Both pixels, and all the other pixels on thedisplay screen, detect the same light intensity based on thesurroundings. At 510, the two pixels are shown after a physical note isplaced on the display screen, covering the two pixels. As shown, thelight intensity on these two pixels has changed. The light intensitychanges on a subset of pixels where the note has been placed. At 515,the two pixels are shown after the brightness has been increased on thearea to be scanned. The results differ based on what has been written ordrawn on the physical note as illustrated in FIG. 6. A portion of thephysical note is written above Pixel 610, while nothing is written onthe note above Pixel 605.

Capture component 52 collects the results from the subset of pixelswithin the scanning area to generate a scanned image reflecting what hasbeen written and/or drawn on the physical note. This generates an analogsignal with a varying voltage (intensity). The converter changes this toa digital signal (0s and 1s). By combining the 0s and 1s for each pixel,a digital black and white image can be built. The scanned image may begenerated in any of a set of supported formats, such as JPEG (JointPhotographic Experts Group), tagged image format file (TIFF), bitmap(BMP), graphic image format (GIF), portable network graphics (PNG), orthe like. Capture component 52 is further configured to log a set ofcoordinates of the pixels related to a location of the physical noterelative to the display screen. In an embodiment, location data may bestored in a storage system, such as volatile memory.

Application component 54 of system 50, as executed by computersystem/server 12, is configured to identify a target application toinsert the scanned image. To that end, application component 54 maydetermine an associated application window that is displayed at thelocation where the physical note was positioned. To make thatdetermination, application component 54 may be configured to query, orotherwise communicate with, the operating system (OS) of the electronicdevice to determine an application which is “active” or “at theforefront” at the specific area where the physical note was placed onthe display screen. If more than one application is open in the areawhere the note was placed, then application component 54 can calculatewhich application is covering the most area, and may select that as thetarget application.

In an embodiment, application component 54 may prompt a user to selector verify a target application. For example, a pop-up box may bedisplayed listing each application that the user currently has open. Inthat case, the user may select the target application to which thescanned image is inserted. In another example, application component 54may prompt the user to verify the application that application component54 has deemed the target application is correct. If a verification isnot provided by the user, then the scanned image will not be insertedinto that particular application.

Presentation component 56 of system 50, as executed by computersystem/server 12, is configured to insert the scanned image into atarget application. To that end, an application programming interface(API) of the target application to the invention would be needed, or theapplication would have some level of OS control to simulate the humaninserting an image. In an embodiment, the scanned image may be insertedwhere the user's cursor is currently located. In a first method,presentation component 56 may execute an “insert image” command on themenu bar of the target application. In a second method, presentationcomponent 56 may also insert the image by adding it to the OS clipboard,and then executing a “paste” command on the target application.

In another embodiment, presentation component 56 may insert the imageinto a target application at a specific location in the application. Forexample, a user may be working in a word processing application which istaking the entire screen, and the user wants the image to be inserted ata location in the application where she has placed the physical note onthe display screen rather than the current cursor location. Tofacilitate this method, application integration and/or API informationbeing transmitted to the presentation component 56 may be required toallow insertion of the scanned image at a location related to a positionof the physical note placed on the display screen, and may vary based onapplication type. In addition, before pasting a scanned image,presentation component 56 may resize the image so that, when pasted onthe target application, it occupies the exact area of space as the areaof the application that was covered by the physical note.

Referring now to FIG. 7, in conjunction with FIG. 2, an implementationof a process flowchart 700 for scanning, as an image, a physical notepositioned on a display screen of an electronic device and automaticallyinserting the scanned image into an application being used on theelectronic device is shown. At 702, a user places a physical note on adisplay screen of an electronic device. At 704, capture component 52scans, as an image, the physical note placed on the display screen. At706, capture component 52 further logs a location of the physical note.At 708, application component 54 determines a target application toautomatically insert the scanned image based on the location of thephysical note. At 710, presentation component 56 inserts the scannedimage into the target application.

Process flowchart 700 of FIG. 7 illustrates the architecture,functionality, and operation of possible implementations of systems,methods, and computer program products according to various embodimentsof the present invention. In this regard, each block in the flowchartmay represent a module, segment, or portion of code, which comprises oneor more executable instructions for implementing the specified logicalfunction(s). It should also be noted that, in some alternativeimplementations, the functions noted in the blocks might occur out ofthe order depicted in the Figures. For example, two blocks shown insuccession may, in fact, be executed substantially concurrently. It willalso be noted that each block of flowchart illustration can beimplemented by special purpose hardware-based systems that perform thespecified functions or acts, or combinations of special purpose hardwareand computer instructions.

Some of the functional components described in this specification havebeen labeled as systems or units in order to more particularly emphasizetheir implementation independence. For example, a system or unit may beimplemented as a hardware circuit comprising custom VLSI circuits orgate arrays, off-the-shelf semiconductors such as logic chips,transistors, or other discrete components. A system or unit may also beimplemented in programmable hardware devices such as field programmablegate arrays, programmable array logic, programmable logic devices, orthe like. A system or unit may also be implemented in software forexecution by various types of processors. A system or unit or componentof executable code may, for instance, comprise one or more physical orlogical blocks of computer instructions, which may, for instance, beorganized as an object, procedure, or function. Nevertheless, theexecutables of an identified system or unit need not be physicallylocated together, but may comprise disparate instructions stored indifferent locations which, when joined logically together, comprise thesystem or unit and achieve the stated purpose for the system or unit.

Further, a system or unit of executable code could be a singleinstruction, or many instructions, and may even be distributed overseveral different code segments, among different programs, and acrossseveral memory devices. Similarly, operational data may be identifiedand illustrated herein within modules, and may be embodied in anysuitable form and organized within any suitable type of data structure.The operational data may be collected as a single data set, or may bedistributed over different locations including over different storagedevices and disparate memory devices.

Furthermore, systems/units may also be implemented as a combination ofsoftware and one or more hardware devices. For instance, program/utility40 may be embodied in the combination of a software executable codestored on a memory medium (e.g., memory storage device). In a furtherexample, a system or unit may be the combination of a processor thatoperates on a set of operational data.

As noted above, some of the embodiments may be embodied in hardware. Thehardware may be referenced as a hardware element. In general, a hardwareelement may refer to any hardware structures arranged to perform certainoperations. In one embodiment, for example, the hardware elements mayinclude any analog or digital electrical or electronic elementsfabricated on a substrate. The fabrication may be performed usingsilicon-based integrated circuit (IC) techniques, such as complementarymetal oxide semiconductor (CMOS), bipolar, and bipolar CMOS (BiCMOS)techniques, for example. Examples of hardware elements may includeprocessors, microprocessors, circuits, circuit elements (e.g.,transistors, resistors, capacitors, inductors, and so forth), integratedcircuits, application specific integrated circuits (ASIC), programmablelogic devices (PLD), digital signal processors (DSP), field programmablegate array (FPGA), logic gates, registers, semiconductor devices, chips,microchips, chip sets, and so forth. However, the embodiments are notlimited in this context.

Any of the components provided herein can be deployed, managed,serviced, etc., by a service provider that offers to deploy or integratecomputing infrastructure with respect to a process for scanning, as animage, a physical note positioned on a display screen of an electronicdevice and automatically inserting the scanned image into an applicationbeing used on the electronic device. Thus, embodiments herein disclose aprocess for supporting computer infrastructure, comprising integrating,hosting, maintaining, and deploying computer-readable code into acomputing system (e.g., computer system/server 12), wherein the code incombination with the computing system is capable of performing thefunctions described herein.

In another embodiment, the invention provides a method that performs theprocess steps of the invention on a subscription, advertising, and/orfee basis. That is, a service provider, such as a Solution Integrator,can offer to create, maintain, support, etc., a process for scanning, asan image, a physical note positioned on a display screen of anelectronic device and automatically inserting the scanned image into anapplication being used on the electronic device. In this case, theservice provider can create, maintain, support, etc., a computerinfrastructure that performs the process steps of the invention for oneor more consumers. In return, the service provider can receive paymentfrom the consumer(s) under a subscription and/or fee agreement, and/orthe service provider can receive payment from the sale of advertisingcontent to one or more third parties.

Also noted above, some embodiments may be embodied in software. Thesoftware may be referenced as a software element. In general, a softwareelement may refer to any software structures arranged to perform certainoperations. In one embodiment, for example, the software elements mayinclude program instructions and/or data adapted for execution by ahardware element, such as a processor. Program instructions may includean organized list of commands comprising words, values, or symbolsarranged in a predetermined syntax that, when executed, may cause aprocessor to perform a corresponding set of operations.

The present invention may also be a computer program product. Thecomputer program product may include a computer readable storage medium(or media) having computer readable program instructions thereon forcausing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network (for example, the Internet, a local area network, awide area network and/or a wireless network). The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and routes the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises andocument of manufacture including instructions which implement aspectsof the function/act specified in the flowchart and/or block diagramblock or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus, or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

It is apparent that there has been provided herein approaches forscanning, as an image, a physical note positioned on a display screen ofan electronic device and automatically inserting the scanned image intoan application being used on the electronic device. While the inventionhas been particularly shown and described in conjunction with exemplaryembodiments, it will be appreciated that variations and modificationswill occur to those skilled in the art. Therefore, it is to beunderstood that the appended claims are intended to cover all suchmodifications and changes that fall within the true spirit of theinvention.

What is claimed is:
 1. A method for transferring a physical note to anelectronic medium, the method comprising: scanning as an image, by a setof photodiodes contained in a display screen of an electronic device, aphysical note positioned on the display screen; logging, relative to thedisplay screen, a location at which the physical note is positioned onthe display screen; selecting, from among a plurality of applicationsbeing currently displayed on the display screen, a target applicationinto which to insert the scanned image based on the location at whichthe physical note is positioned on the display screen coinciding with alocation of the target application; and automatically inserting thescanned image into the target application.
 2. The method of claim 1,further comprising automatically inserting the scanned image into thetarget application at a location of a cursor.
 3. The method of claim 1,further comprising automatically inserting the scanned image into thetarget application at a location based on the logged location such thatthe scanned image is displayed in an exact location on the displayscreen that the physical note was positioned on the display screen. 4.The method of claim 1, further comprising determining a scanning arearelated to the physical note.
 5. The method of claim 4, furthercomprising measuring a first lighting level of pixels of the displayscreen prior to the physical note being positioned; measuring a secondlighting level of pixels of the display screen after the physical noteis positioned; and automatically defining the scanning area consistingof a subset of all pixels based on a difference between the first andsecond lighting levels among the measured pixels.
 6. The method of claim4, wherein the scanning area consisting of a subset of pixels of thedisplay screen is manually defined by a user.
 7. The method of claim 4,the scanning further comprising increasing a brightness of a set ofpixels within the scanning area and collecting lighting informationreflected from the physical note to construct the scanned image of thephysical note.
 8. A computer program product embodied in a computerreadable storage medium that, when executed by a computer device,performs a method for transferring a physical note to an electronicmedium, the method comprising: scanning as an image, by a set ofphotodiodes contained in a display screen of an electronic device, aphysical note positioned on the display screen; logging, relative to thedisplay screen, a location at which the physical note is positioned onthe display screen; selecting, from among a plurality of applicationsbeing currently displayed on the display screen, a target applicationinto which to insert the scanned image based on the location at whichthe physical note is positioned on the display screen coinciding with alocation of the target application; and automatically inserting thescanned image into the target application.
 9. The computer programproduct of claim 8, the method further comprising automaticallyinserting the scanned image into the target application at a location ofa cursor.
 10. The computer program product of claim 8, the methodfurther comprising automatically inserting the scanned image into thetarget application at a location based on the logged location such thatthe scanned image is displayed in an exact location on the displayscreen that the physical note was positioned on the display screen. 11.The computer program product of claim 8, the method further comprisingdetermining a scanning area related to the physical note.
 12. Thecomputer program product of claim 11, the method further comprisingmeasuring a first lighting level of pixels of the display screen priorto the physical note being positioned; measuring a second lighting levelof pixels of the display screen after the physical note is positioned;and automatically defining the scanning area consisting of a subset ofall pixels based on a difference between the first and second lightinglevels among the measured pixels.
 13. The computer program product ofclaim 11, wherein the scanning area consisting of a subset of pixels ofthe display screen is manually defined by a user.
 14. The computerprogram product of claim 11, the scanning further comprising increasinga brightness of a set of pixels within the scanning area and collectinglighting information reflected from the physical note to construct thescanned image of the physical note.
 15. A system for transferring aphysical note to an electronic medium, comprising: a display screenhaving a plurality of pixels, each of the plurality of pixels having aphotodiode and a screen light-emitting diode; a memory medium comprisinginstructions; a bus coupled to the memory medium; and a processorcoupled to the bus that when executing the instructions causes thesystem to perform a method, comprising: scanning as an image, by thephotodiode of each of the plurality of pixels contained in the displayscreen that are within an area associated with a physical note, thephysical note positioned on the display screen; logging, relative to thedisplay screen, a location at which the physical note is positioned onthe display screen; selecting, from among a plurality of applicationsbeing currently displayed on the display screen, a target applicationinto which to insert the scanned image based on the location at whichthe physical note is positioned on the display screen coinciding with alocation of the target application; and automatically inserting thescanned image into the target application.
 16. The system of claim 15,the method further comprising automatically inserting the scanned imageinto the target application at a location of a cursor.
 17. The system ofclaim 15, the method further comprising automatically inserting thescanned image into the target application at a location based on thelogged location such that the scanned image is displayed in an exactlocation on the display screen that the physical note was positioned onthe display screen.
 18. The system of claim 15, the method furthercomprising determining a scanning area related to the physical note. 19.The system of claim 18, the method further comprising measuring a firstlighting level of pixels of the display screen prior to the physicalnote being positioned; measuring a second lighting level of pixels ofthe display screen after the physical note is positioned; andautomatically defining the scanning area consisting of a subset of allpixels based on a difference between the first and second lightinglevels among the measured pixels.
 20. The system of claim 18, thescanning further comprising increasing a brightness of a set of pixelsfrom the plurality of pixels that are within the scanning area andcollecting lighting information reflected from the physical note toconstruct the scanned image of the physical note.